fix(tracing): continue inbound W3C trace context at the ACP boundary - #490

Targets next, and builds on the merged #484 (business↔obs correlation) and #485 (Temporal interceptor) — this is a single logical change, no longer stacked.

Problem

After #484 (business↔obs correlation + Option A) and #485 (Temporal interceptor), an async turn's obs trace still detached from the ingress: the workflow + every activity started fresh traces, unlinked from the event/send API call.

Root cause (proven on infra-staging with per-hop traceparent probes): the ingress traceparent arrives at the agent's ACP server in the HTTP header, but FastACP never extracts it — the active OTel context stays empty. So start_workflow/signal (and the asyncio.create_task background dispatch) fire with no active span, the Temporal interceptor injects nothing, and the workflow/activities root fresh traces.

Fix

Extract + attach the inbound W3C context in the ASGI RequestIDMiddleware, which wraps the whole request (so the background task inherits it via create_task's context copy). Now the interceptor propagates the ingress trace across the Temporal boundary and the workflow/activity inherit it → one connected trace. Fail-open (obs never breaks a request).

Parsing is hardened: it uses the explicit TraceContextTextMapPropagator (immune to OTEL_PROPAGATORS=datadog silently disabling W3C), combines repeated tracestate header lines (W3C/RFC7230 MUST) instead of collapsing to the last, and excludes inbound baggage so attacker-controlled keys aren't pulled into downstream context. traceparent is single-valued, so trace linkage is unchanged.

Verified (infra-staging)

Trace 12c6290e…: 347 spans across all 4 services in one tree — gateway POST /agents/{id}/rpc → agent ACP → RunWorkflow → HandleSignal:receive_event → RunActivity → egp/identity/DB. Per-hop probes confirmed traceparent went from <none> → the ingress trace at every hop (ACP + worker). (Requires the control-plane header-forward fix, scaleapi/scale-agentex#396, to deliver the header.)

Tests

tests/test_trace_context_extraction.py — extract-makes-inbound-active, fail-open on missing header, safe detach, repeated tracestate combined, and inbound baggage not extracted. ruff clean.

🤖 Generated with Claude Code

Greptile Summary

The PR continues inbound W3C trace context at the ACP request boundary so asynchronous Temporal dispatch inherits the ingress trace.

• Adds fail-open extraction and lifecycle-safe detachment of inbound traceparent and tracestate.
• Preserves repeated tracestate header values while excluding inbound baggage.
• Adds focused tests for extraction, repeated headers, baggage exclusion, and safe cleanup.

sequenceDiagram
    participant Gateway
    participant ACP as ACP Middleware
    participant Handler
    participant Temporal
    Gateway->>ACP: HTTP request with traceparent/tracestate
    ACP->>ACP: Extract and attach W3C context
    ACP->>Handler: Run request under ingress context
    Handler->>Temporal: Start workflow or signal
    Note over Handler,Temporal: Async dispatch inherits attached context
    Temporal-->>Handler: Dispatch accepted
    Handler-->>ACP: Response
    ACP->>ACP: Detach context safely
    ACP-->>Gateway: HTTP response

_{Reviews (4): Last reviewed commit: "fix(tracing): parse ACP ingress headers ..." | Re-trigger Greptile}